XM 004 812 



AOTHOE 
TITLE 

INSTITOTION 

EEPOB^T NO 
POB DiTE 

vHOTE 



Besei, ^Ronald ^ * ' * _ / 

Mastery-Learning Decision Variables*', 
Southwest Regional Laboratory for Edap?itionafl 
Research and Development^ Los 'ilamitdjSr Calif.. 
SirRL-TH^5-71-08 
18 Aug 71 

24p.; For related documents^ see TM 00^1 811 and 
814 . . 



EDRS- PEICE 
DESCRIPTORS 



IDENTIFIEES 



ABSTRACT 



HF-$0.76 HC-$1.58^Plus Pp.stage 

♦Criterion Referenced Tests^ l^ecision Making;. Grwjps; 
Individual Tests; Instructional Design; .*Mathematlcal 
Models; *Matrices; Probability; Psychometric^; 
Statistical Analysis; Test Interpretation; Test 
Reliability; *Test Results 

Learning Hierarchies; *Mastery Learning Test Model 
(Eorick and Adams) ; Mastery Tests 



The Mastery-Learning test model i?. extended. Methods 
for estimating prior probabilities a^e described. Tie use of ^n 
adjustment matrix to transform* a probability ^of mastery measure and 
empirical methods for estimating adjustment matrix parameters are 
derived* Adjustment matrices are interpreted .as indicators of 
instructional ef:fectiveness' and as evidence of the existence' of 
learning hierarchies. Two decision variables are considered; 
probability^^^i>.-i4stery for an individual and proportion in mastery 
for. an instructional group. Discussion of the reliability^ / 
complexity J and interpretability of these decision variables ^d 
comparison/with decision variables for other test models is a^so 
included/ (Author/DEP) , 




* * * * * * * :ft * * :|c :|c :|c a|c :|c a|c a|c a|c a|c 9^ 3|c )^ 3|c 3|c 3|c 3|c 3|c 3|c 3|c 3|c a|c 3^ 



Documents acquired by SSRIC include many informal unpublishd^ * 

materials not available from other sources. EglC makes every effort * 

to obtain the best copy availably nevertheless^ items, of marginal. * 

reproducibility are oftjsn encooift^red^and this affects the gualityX * 
of the microfiche a'nd hardcopy^ reproductions ERIC makes available 
via t'he ERIC Document Reproduction Service (EDRS) . EDRS is not 

* responsible for the gualit^y of the original document* Reproductions * 

* supplied by EDRS are thfi/fiest that can be made from the original. * 

:|C 3(r :|C 4t 9«t 3|C 3)C 3tC 3(C 3(C 3(C 3|C 3(( 3|( 3|( 3|( 3|( ^ ^ 3|C 9|( 9|( 9|( ^ 3(C « 3tt * * * 




SOUTHWEST REGIONAL LABORATORY 
TECHNICAL MEMORANDUM 



OATE: August 18, 1971 
xMO: TM 5-71-08 ' 



U$ OCFAtTMCNTOFHEALTH, 
EDUCATION 4WELPAJ16 
NATIONAL INSTITUTEOF 
EDUCATION 

OUCEO EXACTLY AS RECEIVED PROM 
THE PERSON OR ORGAN»2ATrON ORIGIN 
ATlNOa POINTS OF VIEW Oft OPINIONS 
STATED DO NOT NECESSARH.Y REPRE 
SENTOFFrClAL NATIONAL INSTITUTEOF 
EDUCATION POSITION OR POLICV 



TITLE: MASTERY-LEARNING DECISION VARIABLES 



AUTHOR: Ronald Besel 



ABSTRACT 

The Mastery-Learning test model described in TM 5-71-04 is extended. 
Methods for estimating prior probabilities are described. The use 
of an adjustment matrix., to transform a probability of mastery measure 
and eiiq>iticalWthods for estimating adjustment matrix parameters are 
derived. Adjustment ^trices are interpreted as indicators df .instruc- 
tional effectiveness and as evidence of the existence of learning' ' 
hierarchies*. Two decifsion variables are considered; probability of 
mastery for an individual and proportion in mastery for an instructional 
group* Di.scuasion of the reliability, complexity/ and interpretability 
of these decision vari^ablcis ?nd comparison with decision variables for 
other test models is also included.. ^ 




TbU docua«nt ti tnt«nd«d for lnt«rn«l tt«f f tdlitrtbution ind u<«. PeraUtton co reprint or quote ^ron thtt, working 
doCQii«nf« wholly or In ptrt, ihould be obtslned fro« SWU., 11300 U Ctenege Boulevard , Inglewood, C^ltfo^ntV, 9030A.* 



MASTERY-iEARNING DECISION VARIABLES 



Introduction * ^ ' 

This .paper focuses on the analysis of test data by a mastery- 
learning test model. The inputs for the test model are tl)e responses 
of individuals to test items; these responses are classified as either 

^ correctior incQrrect. Th^ outputs of the test model are jSfflled deci- 
'Sion variables. The' test mddel described here is algorithmic, i.e., a . 
mathematical model *is used to comp^g_ii^^^]^±cal values for the decision 
variables. It is an extension of' the Ka^te:^ T^ear^r^ and 

^H^dams, 1970) as described in TM 5^74-04-^esel, 1971). Two decision 
variables are considered: probability of mastery for an individual and 

' - ' / / 

roportion in mastery for^an instructional j group. 




Symbols and Notatior 

Variables will!U>^" represented by capital letters.' If a variable 
I ' ' * 

represents a vector! the elements of the vector will be represented by 

the equiv^alent' lowdr case symfedl, 

' Lower case letters and numerals will be, used as subscripts. The- 

subscript (i) refers to a test. item, (j) to an |.ndividual, (k) to a 

performance measure for an objective. If a relationship among variables 

dbes nqt fefer to a. particular test item, individual or objective, the ^ 

corresponfiing subscripts will be deleted from th6 symbolic representation 

of .the. relationship. The ^^>tl6script^(j) will be used to refer to a second 

test itei in an e*xpressrfon relating two composite tests. 
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.Capital letter P will refer to a probability • Functional rdlation- 
sYi-vps will be denoted by enclosing the' independent variable in parentheses 
• (e.g., ?iX/lO will represent the conditional probability of a response X 
given that- the individual is in the mastery (M) Atate). The ^et theoretic 
notation M will be.u^ed to denote not being in the\ mastery (M) state, 

The Mas tery- Learning Model 

The mastery-learning model (TM 5-71-04) assumes ^at a fe^st measures 
proficiency with respect to a single skill and that ,ther^ are only two 
states of proficiency for that skill. Each individual 'tes^d is in 
either the mastery (M) or non-mas tery(M) state at the iiime of testing. 
The only true scores are assumed to b^ 0 and K (for a K-item test); all 
intermediate scores are due to measurement error. 

There ard two. 'classes of measurement errors: wrong respia^ses by 
individuals in;the mastery state (p errors) and correct responses by 
individluals in the non-mastery state (a errors). 

or^ = the probability that an individual in the M state will give 
a correct response to the i^^ item, 

3^ = the probability that an individual tin thjg M state will give 

th "^v-^^ 

an incorrect respionse to the i item. ""PX 
The and parameters are as.sum^d to have ,1^4 values which are' 
characteristic of the test. Bnrick aitd Adams' model has been mpdifiied 
to permit item parameters /rathef than single or' and p test parameters^ /' 
The assumption is still made that arid p .have the' same .va^.ue fpr : 
every individual belonging to a common instructtbhal gjroup; 



Let, 



Xj,j repfresent the response of individual j to item i, 



[1 if a correct response is given 
1 0 if an incprrect^*response is given 



(1) 



'^Ij ' ^2j ' ^3j 



'^Tj , ' (2) 

represent the response vector- for individual j ■ 



S. = 



rt^present the test score for individual j 



(3) 



Sequential analysis of the item responses' will be asstimed in the deriva 
tlon of the mastery- learning model* -The (j) subscript will be delated 
to siii5)lify the notation. . ^, 

For any individual tested the following conditional probabilities 
are associated with his response* to the first Item: r ' . 

PXx^ = 1 / M) = L-p^ . 



P(Xj^ - 1 / M)/ = CYj 

;p(x^ = 0/ M) = l-a^ , • 

!flie probability. that a response indicates a particular state can 

' ' ' i ' ' ' 

b(B coii5>uted using Bayes forrmila: 



(4)' 
(5) 
(6) 
(7) 



s ^ 
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where , 

PR^ represents the prior probability .of the mastery statje, 
P(x^) represents the prior or expected distribution of x^^ given by/ 
P(x^) = PRM.P(x^ / M) + Cl-PRM]^P(x^ %^)( ' ; (9)' 



For a correct response. 



PCM / x^ = 1) = 



PRM ) 



For an incorrect response. 



(10) 



P(M /^x^ = 0) = 



PRM. op 



PRM' 



•3^ -f Cl-PRM3.[l-a^] 



(11) 



ilethods for estimating prior ^probabilities will be discussed later in 
this paper. ^ - - 

•rtt^ conditional probability of mastery based oji the first item 
respbi\se^(equation 8) is used as the ,prior probability for. the second 
item response. ' . 



,P(M7 , x^) = 



P(M/xp.p(x2/M) 



P(M/xp.p(x^/M) + Cl-i'(fl/xj^)].p(x2/M) 
Substituting equations (8) and (9) for P,(M/xj^) yields; 

' PRM [P(x^/M)..p(x^/M)i 



(12) 



P(M/x^ , xp = 



y2 PRM.CP(xpM).p(x2/M)] -^ [l-PRM] .Cp(x^/M) .p(x /M) } 



(13) 



ERIC 



-This procedure can be repeated, sequentially computing, the conditional 
.probability of mastery given the i^ item response, with the prior 
probability based oA ttie^ previous responses. For a K-item test: 

P(M/X) = ' . ^ . .(14) . 



For' any length test, ' ^ ■ ■ ' ~ ^ 

P(M/X) = 1-P(M/X) . (15) 

If average value? of and are estimated rather ^han item parametera 

equation (14) becSmesr^ • . . . ' < • 

r 

Estimating the Proportion of Students in the Mastery State 

- ' jf • ■ 

The proportion-in-mastery for a group of -students can be estimated 

from the observed mean score fbr the group. The foIlowiVig deriv ation of 

the^ relationship- between mean score and proportion-ih-raastexy makes two 

independence assumptions: ' *' 

(1) The responses of a student to eetch test item are independent 
of the Responses of all pther, students in the group. 

(2) An individual's responses to- the separate items on a test 
can.be treated as. a sequence of independent' trials. ' 



6- 



Let, E(S^) represent the e:q)ected score for the individual and. 



N 

S S 

5=1- 



(17) 



represent the observed sanpl^e mean for a group of N students. From 
Che first assuiiq)tion, the expected value of the observed sanqile mean 
is: 

(18) 



/ 



■E (0) = ^ . 



N 

r E (s,) 
j=i 



For an item with parameters {oi^ , 3^) , 

= 1-pj^ for the individuals in the mastery state. (19) 



E (x^) 



E (x^) 



M 



M 



= Of, 



for the (N-N ) individuals in the non-mastery 
state. ™ (20) 



From the second assumption the expected scores for individuals in the M 
and M states are: ^ 
K 

= S (1-3,) for the N individuals in M state. (21) 
M i=l ^ 



E (S) 



E (S) 



K 

= S a, for the N-N individuals in M state. (22) 
M i=l ^. 



The expected value of the observed sample mean is then. 



E (U) = - 



1 
N 



N E(S) 
m 



K 



M 



+ <N-N^).E(S) 



M 



•N^ . S (1-3 ) + (N-N„). S a, 
i=l ^ i-1 i 



(23) 



(24) 
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Define the' proportion- in-mastery to be^ 

, N ^ - ' ■ ' 

MP^ = / ' , , (25) 

and^lie estimated value of proportion in ma&tery for a particular group 
to be GMP. ^ \ , ' 

Then, ^ 

E(U) = MP. S (1-3, -a.) ■ + --a/ " ' (26) 

i=l ^ ^ i=l ^ 

Using the observed sampl,e mean, U, as an estimate" of E(U) and solving 

for GMP yields: * . . ' 

K ' ' . 
' U - S Qf^ , - ■ 

GMP = ^ _ ' (27) • ■ 

2 (l-?,-a,:) ; . ' ^ ' 

if average values of or and g are employed rather than item values, this 
relationship sinqjlifies to, 

U /K - a 
1-3-a, 



GMP, ^ ' . (28) ' 



An estimate \)f the proportion of students in the mastery state can 
be used for two, distinct purposes:, (1) it can be used directly-^s a 
decision variable by the evaluator to judge /the effectiveness of an, 
instructional unit or by the teacher in sel^ectirtg an appro'priate strategy 
£or^ remedial instruction. If only a small percentage of Jn" instructional 
grotip has achieved tj4stefy, review or second instruction for the entire^ 
group may be warranted while tutorial assistance for non-masters nnay be 
preferred If el lar^e percentage has achieved mastery; (2) it can also be 
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used to estimate prior probabilities needed in the coTi?)Utati0n of 
probability of mastery. * ' / * * 

' : ■/ : ■ : , - 

Prior Probability Estimates " 

Two general classes of prior prd^ability estimates can be us^d in 
the computation of probability of riiastery. The first includes all 
methods which assign the same prior probability to each individual in 
a group. The second class includes all methods which use other test 
data pbirained from an individual to estimate a ''personalized" prior 
probability. 

^ ' The proportion of students in mastery estimate can be used directly 
as a Class '1 prior probability. Each student is assigned the same, prior 
probability of mastery, GMB; as the group mean score increases, the prior 
probability estimate increases for each member of the gfoup. ' 

Emfick and Adams (1970) suggested a different type of Class 1 



estimate. The anticipated instructional effects, i.^., the anticipated 
'proportion of students in mastery after instruction, can be estimated 

^ / 

from relevant past experiences. The performance of a similar group of 
.students ^during the previous year may be releverit if the instructional 
activities are comparable. This approach may be best suited for "small" 
liistrdctional groups and replicated instructional treatments while the 
GMP estimate, can be used for "first time" ii^structibnal treatments 
applied tb sufficiently large instructional groups. ^ „ 

The only tj^e of Class 2 estimate which will be considered • here is ^ 
based' orv test data which have been analyzed in terms of probability of 
mastery of some performance objective. The only requirement is that 
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the responses of an individual to the test items currently being analyzed 
be iti'dependent from the responses ten. test items used for the prior prob- 
-ability estimate, ^Thus, the- test used may be a parallel test for the 
same perfotmanpe objective, it may\be a concurrently administered test 
for a diffetent performance objective ox a previous measure of a per- 
formance objective. 



Adjustment Matrices 

It wxll generally be desirable to "transform a probability of mastery 
measure in order to use it as a personalized prior probability estimate, 
A linear transformation of a prior performance measure to estimate a 
•personalized prior probability will bre repte.sented Uy^an adjustment 
matrix, , • 



c(k) 



^0,1 



(29) 



.A ... is the adjustment matrix used to estimate, a. personalized .prior 
^probability for a current performance ifte'asure .(cX from performance measure 
(k)V The personalized prior .probability: ;for measure (c) is then. 



l-PRM, ... 












PRM, ,' . 




"1.0 , 


%1 







(30) 



Where PM. , is the probability of mast67^ for the J in3iv|.dual for 
performance measure (k), : 



The ^ ^arameterV in aij ai^^stiaent. matrix are cprtdipional probabilities.^ 

For exanqjle , ^ ^ V . ^ • 

Q is the expected probability that the individlual will bfe in ' 
the mastery state for the current^ objective pn the condition 
that he was in the *, non-mastery state for the prior objective. 

Since the a - parameters are probabilities, * ' 

' CjK J 

^0,0 h:o h ^ / ^^^^ 



and ' 



Adjustment matrices have educational interpretations folr particular 
choices of prior performance measures. If the prior performance measure 
is a pretest' In the ?opi of a paratl-el measure of . the current perfprmaiice 
ineasure, the adj^ustment matrix can be interpreter's a ^representation of 
inVtructional effectiveness. The % < value for "an instructional ef f ec- • 
tiveness *matrix should be very clcJs^ to 1;0 rsince it can usually be 
assumed that instruction wij.1 not lead to a transition ^ from m^^ stery to^ 
a non-master|y state. The a^^ Q^;y:alue is,, for the^case of a^^ ^ equal to 
1.0, a'useful Index of instructional ef fec^veness. 

!rf *^ hierarchial relationship be.tween two^!:i>etformance measures ! 
exists, a second typ^e of adjustm^t matrix with a meaningful interpre- 
tation can tie constructed. Suppose objective, (k) miisf be*inastered before ^ 
th^current objective,, (c)-, can be mastei;ed. The adjustment matrix, 
A^^j^ J should have an a^^ q value equal to zerp.* 



UiAiased Prior- Prdbabill tie's * 

A prior probabtUty estimate, for'* cutrent performance measure (c) 
Based on performance measure (Ic). defined to be- unbiased if; \ >. 



GMP, 



■ '(33) 



■ , . is the 'proportioji-in-mastery. for the current performance 

m^asur^. /, ' • . , ^ ... 

A graphical representation of^an adjustment matrix facj^litatejs the 

* ' • . • ^ • " « 

explanation of this definition (see Figure:!).* 



1.0 



Prior 
* Prob'ability 



PRM 




ProSabillty' of "Mastery 



Figure 1] 



1.0 for Measure k 

Adip^tment matrix for estimating a prior probability for current 
D«formance measure (c) "from a probability of m'aStery value ^. 
Computed for' measure (k). ^ V ; » . » 

An adjustment foatrix is ^presented graphically by a straight line 

which has an intercept equal to a- « and a slope* equal to* (a- - -,^a- 
J / i 1 5I " 1 5® 

Equation' (33) results from substituting GMP, for PM^ , and GMP for 
' . / ° k j.,k c 

PRM ^ 



j,c(k): 



A Correlational Method for Estimating Adjustment Matrix Parameters 

The correlation between two dichotomous Measures can be computed, 
using the formula for ^ Phi coefficient: ~ 



Pi 1 -Pi. - PiPo 
Phi = -lii— i LI. 



(34) 



where and represent .the marginal .probabilities of b^ing in* 

, the (1) state for the first and second measure respectively and 
*^ 

p - - represents the conditional probability P(?^ = 1/x- = 1). 

A contingency table can be used to display the correlation betwe^ 

two dichotomousNneasures, ; ^ 

Measure 1 
*P, 



.1-^1 



i-p 



2 • 



Measure 2 



p . 
0,0 


^o.i' 


p 

%o- ► 





Figure 1. Contingency table relating Variable 1 to. Variable 1. 

TKfe following relations amottg conditional* and marginal probabilities 
will be used l4 later derivations: 



0,0 


•f 




"0,1 




^1,1 


p' = 

- 2 




. p. 






. p. 

J 



= 1 



1,1 



1,0 



(35) 
(36) 

(37) 
(38j) 
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Two contingency tables are 'of particular interest: the contingenay 

7 . ■ - . 

tables for an adjustment matrix and for. relating observed response to 

true state, ' ' ' 



Current 
Iteasure 











, Prior .Measure ' 




1-MP, 
k 




1-MP 

c 


^0,0 




MP 

c 


' "1,0 


""1.1 



figure 3. Prior Measure.- Current Measure 
Contingency Table 



True State 











1-MP 


MP 


Incorrect (0) 
Observed ^""^^i 




^ ■ -r 

^i 


Response 

Correct' (1) 




1-3^- ■ 









' Figure 4. , True State Observed Response , - . \ c - • 
. Contingethcy Table for. a single .'test i'tem. 

The symbol PC^ will be. used, to represent the probability of 'i 
correct response to item (i). if, ^ respondeiffc is' randomly selected f torn 
the population, , ' . ' • - . " . 

Observed score will be Refined tO be' related to true score for 
measure^ (k) by the following equation: 



+ e 



i,k 



(39) 
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. .where tj^ is the true score; " ' . 

for an individual in the non-mastery state, 

1 for an individual in the mastery state, 
/ 

' r 

and e. , is the measurement error; 



i if* a non-master gives a correct response, / 
• 1 if a master gives an incorrect ^response, ^ 
0 otherwise^ ' 



Sipce x; , and t, are dichotomous variables assuming only the values 
and on^. 



zero 



^and, / ■ » ^ 



/ / 

( /' 

(40) 
(41) 



The' correlation between true score and observed score is computed using 



equation (34) . - ■ > 



MP (1-P ) MP . -JC' -".- 
^(x , , t) = ^ / ^'^ • (42) 

g^i-.k ^MP^ (1-MP^) V^^i.k <^-^^i,k>\ . 



Using rela^tion (37) to exprjass PC^ ^ in terms of MPj^, 

^>)(x^^, , t^) = ^ ^ . (43) 



^hich sin5)llfies to, 

2>, 



(44) 



RJC 
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frpih the general relatidnship for a* correlation, 

it cSh ijeadify be seen that the ninnerator of (44) is the covariance 

■J 



xyhere'Y. , *iK^efined to be. 



(46). 



it can be' shown that the covariance of any two dichotomous items is,. 
o a(x^ , x^^ = a(t^ , V>i,l-Yj,2 ' 

where 't- • is the true ^score for measure t, t^ is the true score for 

**' • 1 " ' X ^ ^ . 

measure 2 &nd x. , ^- are observed responses to items selected from 
mea^Ure& L and 2 respectively. • / c . '* 

Relations for the covariance and correlation of composite variables 
(see equation 3) are derived directly fr^ item relations. 

Define to be the average value^ of Y£ 







i ^ 




\ 


^ i=l ^ 




Kl 


K2 




= E ■ 


E cr*(x 
j=l ' 




Kl 


^K2 




= E ■ 


-S- CT(t 




i=l 





i,l ' ''j,2^ 



• 'j,2 
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•16-/ / 



/ 



TCI 

i=l -'^ 



K2- 

S V 



.■ Kl 



(48) 



. Kl 
S' Y. 
ii=l 



= a (t^) (Kl.Yi) 



(4a) 



For Classical Test Theory, the^following correction for attenuation ^ 
fonnula is used"; to estimate the correlation between true scores from 
the correlation between observed .Scores (Lord and Novick, 1968). 




(50) 



It will be shown that this rel^tlXn holds for the mastfery learning model 
as well. 

Substituting (45) itlfc/ (50). yields 



/to) .= 



4) .cT(t^) a(t^) 



(51) 



,en using relatlA)ns (48) and (49) 



t^) (Kl-Y^) (K2.Y2) a(tp ait 2) 



■(t^) (Kl-Y^) a^t^) (K2.Y2) 
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7 aCt^ ^'^2^"' 

which is an identity. 

For an adjustment matrix, 



(52). 



■K 



^(t,)a(tj^). 



(53) 



Solving for a^^ yields * 



MP 



(54) 



MP, 



Using GMP, to estimate MP, , GMP to est^Linate MP and the relation between^ 

k ' k ^ c • *; c ' 

a(t^ ,^tj^) and a(S^ , Sj^) leads to- 



rTM 



I 



(?5) 

! • 



GMPj^ (Kl*Yi) (K2*Y2> 

If the adjustment matrix is restricted to be unbiased, equation (33) 
can -be used to solve for the a^^ ^ parameter* ^ , 




1 - GMP, 



a<S^ ; S^) 



(56) 




_ ( 1 - GMPj^)-(Kl.YiJ (K2.-Y2) 

Formulas (55) and (56) can be used to Obtain adjustment matrix 
paijameters empirically. Methods for estimating the a^, parameters, 
will be ^iscussed in a subsequent paper. 
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^ The Adjustment Matrix for Parallel Testa 

Two mastery tests are parallel if the correlation between true 
scores for -the tests is equal to 1. The correlation between two 
dichotomous ineasur^^ can be unity only if: 

, ^1- = ^2 • ' . . ^^^^ ■ 

which means that the prdpprtion-in-mastery values are equal, and, - 

- '^1,1 * (58) 

From'eguation (37), it can be seen that perfect correlation also in?>lies 
that: 

In the derivation of the Master Learning Model, sequential analysis 
of item responsjes w^js as^timed^W be prbper. This is equivalent to, the 



assumption that one (or several) items can, be used as a prior measure^- 
with an adjustment matrix satisfying (58) aftd (59*)*— in coicputing prob- 
ability of mastery given one additional item response. The assumption 
reduces to the requirement that the test items are paralle^l measures. 

It should be noted that the a^, p^" parameters and the item diffi- 
culties (PC^) need not be equal, for items to be parallel. The asstjmption 
of parallel items is. thus less restrictive than* is the case for Classical 
test theory (Lord and Novick, 1968)* • ^ , 

Reliability, 'Complexity, and Interpretahility 

The following properties influence the utility of decisions variables 

as inputs to a decission process: . . 

» 
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(a) Reliability: the measurement property of being fepeatabje,. 

A second, testing results^in the same rank-ordering of individuals 
(or groups) or the same classification into categories. 
» (b),Con5)rlexity: to make effective decisions requires the derivation 
i and application of complex, decision rules ♦ A decision procedure 
tihich is difficult to explain to individuals responsible for the 
decisions made will be labeled "coniplex^^ as well as procedures, 
requiring lengthy computation. ^ 
(c) Interpretability: the decision variable^has an intuitive or 

easily understood ^meaning for the deciaidn maker* 
It is much less con5)licated to derive an objective basis for comparing 
the reliability of decision variables than it is for the cofnplexity and 
interpretability properties. This alone is not a valid reason for selefcting 
decision variables solely dti the ba$is of reliability. , The rel-ative impor-- 
tance Of each ,of these properties for effective decision making must be 
considered; this will depend both upon the type o.f decisioti being made and 
the characteristics of the decision maker. • 

Comparison with two distinctive tests models will be used in discussing 
reliability, coii?)lexity, and interpretability. 
Test Mo4el 1: Standardized-Normal 

(a) * Individual . Student Decision Vafriable— standard (z) score 

(b) Group. Decisidh. Variable—group mean score 
Te^t Model 2: Criterion-Referenced ' ' 

(a) Individual Student Decision Variable—raw or percentage 
\ correct a<iore 

(b) GrouJ) I)ecision Variable—percentage exceeding criterion score 
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Reliability indices for the Standardized-Normal Test Itodel attempt 
to assess whether the raxdc-orderitig of individyals is preserved with 
repeated testing. Preservation of rank-order is important for applica- 
tions requiring correlational analysis* and some selection problems. X 
, Criterion-Referenced Test Model is more frequently used for different 
types of applications (performance assessment- and placement); A reli- 
ability index which assesses the repeatability of classification into 
several (usually two) categories may be most appropriate for these 
applications. ^ 

The appropriate reliability .index is thus dependent upon the pAr- 
ticular application employing the decision variable. Reliability assess- 
ment for the Mastery-Learning Model i^s complicated by the ^possibility of 
having a liumber of measures available which are potential prior measures. 
A different reliabj.lity coefficient can be computed for each choice of a 
prior pe^rformance measure. For those who accept the notion of multiple 
indices of test validity, this coiq)lication should impose con5)Utational 
rather than conceptual difficulties. It should be possible to select 
prior measures which in5)rove both the reliability and'^alidity of mastery 
learning decision variables for those applications for which Criterion- 
Referenced analysis is currently used. ' ^ / 

As defined »in this paper, assessing the 9on5)lexity of a decision 
variable requires the prior exi'stance"" of decision rules which have been 

evaluated in* terms of effectiveness. A simple decision rule which is 
. *• 

adequate in one si^ation may be ineffective in another. Theoretically 
derived decision rules, using standardized scores tend to be complex 
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(Cronbach and Gle^er, 1965); they would be quij:e difficult to explain ±o ^ 
the ^0^^|^^thematician8 and may require cons.lderable computational power • 
' Decision procedures for the Criterion- Referenced Model are degraded by 
the absence of an assumed distribution' of true scores • This tends to 
both complijCate the decision procedures and reduce its effectiveness 
(Besel, 1971). 

It may be possible to use approximate decision rules which satisfy 
the complexity criterion and achieve acceptable results but the interpret- 
•ability of the results may suffer. Arbitrary' choice of criterion-scores 
for Criterion-Referenced llodels an<J cutting scores for Standardized-Normal 
^k)del may simplify decision procedures at the expense of understanding the 
effects .of, altering "^tl)e choice, ^ ' * . 

Theoretically based decision rules can be^ derived for the Mastery^ 
, Learning Model which are relatively sjLjiqjle to explain and calculate. For 
interpretabilit'y, they depend only on the concepts of subjective probability 
and expected lo>is.^ Bnperical verification of these claims remains to be 
attempted, • ' . • , ^ 
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